Modern mobile terminals, such as mobile phones, are often equipped with digital cameras. Such digital cameras are often used to record still images and/or video images. Many mobile terminal cameras, as well as other imaging devices, use image sensors that record images using what is known as a “rolling shutter” technique, as opposed to “global shutter” technique. For example, a commonly used image sensor that uses a rolling shutter is a complementary metal oxide semiconductor (CMOS) image sensor.
With a global shutter image sensor, all of the pixels of the image are captured at the same time. However, with a rolling shutter image sensor, the image sensor sequentially charges rows of detectors, waits a short exposure time, and then reads the exposure information row by row. In other words, rolling shutter sensors result in images where the actual time of exposure is not uniform for the whole image area. Therefore, rolling shutter image sensors may result in images with some amount of distortion if some object in the image is moving relative to the image sensor, including situations where the object is moving and situations where the sensor is moving. Although rolling shutter sensors sequentially expose all of the rows relatively quickly, for very fast motion, such as the swinging of a golf club, the object may move a human perceptible distance between the time when one row of pixels is exposed and the time when another later row of pixels is exposed. In general, the faster an object is moving relative to the camera the more distortion there will be in the image of that object. This can be true for both still images and for individual frames in a video.
For example, FIGS. 1a and 1b each illustrate a frame from a video of a golfer swinging a golf club. FIG. 1a illustrates how the frame 100 may typically look when recorded using a global shutter image sensor. Since the global shutter image sensor exposes the whole image at the same time, the image of the golf club 110 is not distorted by the image sensor. In contrast, FIG. 1b illustrates how frame 150 may typically look if the video were recorded using a rolling shutter image sensor where the rolling shutter image sensor exposes rows of pixels sequentially from top to bottom. Since the actual exposure time at the top of the image is earlier than the exposure time at the bottom of the image, the fast moving golf club 160 appears noticeably distorted. More specifically, since the golf club 160 is moving very quickly relative to the camera from the left side 151 of the image 150 towards the right side 152 of the image 150, the golf club 160 moves slightly more to the right each time the image sensor exposes a new row of pixels. Thus, the golf club shaft 161 has a curved appearance in the frame 150, curving towards the right side 151 of the image 150 as golf club shaft 161 gets closer to the bottom 153 of the image 150. Although the golfer's body 165 is also moving relative to the image sensor, in this example, the speed of the rolling shutter action of the sensor is high enough relative to the speed of the motion of the golfer's body 165 and hands 166 such that the golfer 165 does not appear noticeably distorted. However, the speed of the golf club 160 is much faster than the speed of the golfer's body 165 and is high enough so that that the image of the club 160 is noticeably distorted. This effect is referred to as motion skew.
Despite the distortion that can result from a rolling shutter sensor, many rolling shutter sensors, such as CMOS sensors, have advantages over other types of sensors. For example CMOS sensors are usually more economical than most other types of comparable image sensors. Furthermore, many rolling shutter sensors can capture and read out the image data faster than a global shutter image sensor since the rolling shutter sensor typically reads out the data captured by one row of pixels while another row of pixels is being exposed, while a global shutter sensor exposes all of the rows of pixels and then must read out all of the captured data at the same time. As a result, video cameras using a rolling shutter image sensor may be able to capture more frames per second than a camera using a global image sensor. Thus, a solution is needed to fix or at least reduce the image distortion that can be caused when a rolling shutter image sensor is used to capture an image of an object moving relative to the image sensor.